An Empirical Comparison of Mutant Selection Approaches

Mutation analysis is a well-known method for measuring the quality of test suites. However, it is computationally intensive compared to other measures, which makes it hard to use in practice. Choosing a smaller subset of mutations to run is a simple approach that can alleviate this problem. Mutation operator selection has been heavily researched. Recently, researchers have found that sampling mutants can achieve accuracy and mutant reduction similar to operator selection. However, the empirical support for these conclusions has been limited, due to the small number of subject programs investigated. The best sampling technique is also an open problem. Our research compares a large number of sampling and operator selection criteria based on their ability to predict the full mutation score as well as the consistency of mutation reduction ratios achieved. Our results can be used to choose an appropriate mutation reduction technique by the reduction and level of fidelity to full mutation results required. We find that all sampling approaches perform better than operator selection methods, when considering ability to predict the full mutation score as well as the consistency of mutation reduction ratios achieved.